Query Logs Alone are not Enough

The practice of guiding a search engine based on query logs observed from the engine's user population provides large volumes of data but potentially also sacrifices the privacy of the user. In this paper, we ask the following question: Is it possible, given rich instrumented data from a panel and usability study data, to observe complete information without routinely analyzing query logs? What unique benefits to the user could hypothetically be derived from analyzing query logs? We demonstrate that three different modes of collecting data, the field study, the instrumented user panel, and the raw query log, provide complementary sources of data. The query log is the least rich source of data for individual events, but has irreplaceable information for understanding the scope of resources that a search engine needs to provide for the user.